<html>
<head><meta charset="utf-8"><title>crater is timing out · t-infra · Zulip Chat Archive</title></head>
<h2>Stream: <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/index.html">t-infra</a></h2>
<h3>Topic: <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20is.20timing.20out.html">crater is timing out</a></h3>

<hr>

<base href="https://rust-lang.zulipchat.com">

<head><link href="https://rust-lang.github.io/zulip_archive/style.css" rel="stylesheet"></head>

<a name="246667109"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20is%20timing%20out/near/246667109" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Joshua Nelson <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20is.20timing.20out.html#246667109">(Jul 21 2021 at 00:34)</a>:</h4>
<p><a href="/user_uploads/4715/NcAhtLPkojQkq3QcBrAIO59E/image.png">image.png</a></p>
<div class="message_inline_image"><a href="/user_uploads/4715/NcAhtLPkojQkq3QcBrAIO59E/image.png" title="image.png"><img src="/user_uploads/4715/NcAhtLPkojQkq3QcBrAIO59E/image.png"></a></div>



<a name="246667695"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20is%20timing%20out/near/246667695" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Joshua Nelson <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20is.20timing.20out.html#246667695">(Jul 21 2021 at 00:45)</a>:</h4>
<p>still timing out :/ not sure what's wrong</p>



<a name="246668241"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20is%20timing%20out/near/246668241" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20is.20timing.20out.html#246668241">(Jul 21 2021 at 00:55)</a>:</h4>
<p>seems to be generating a report; that currently causes the requests to timeout until it's done (not actually confident why; might be we're holding a database lock, but I don't think we do. Could just be due to CPU being pinned or something).</p>



<a name="246668298"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20is%20timing%20out/near/246668298" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Joshua Nelson <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20is.20timing.20out.html#246668298">(Jul 21 2021 at 00:56)</a>:</h4>
<p>oh that's actually good news, that means the PR I wanted to look at is almost done :)</p>



<a name="246668308"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20is%20timing%20out/near/246668308" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20is.20timing.20out.html#246668308">(Jul 21 2021 at 00:57)</a>:</h4>
<p>CPU is mostly 90% in iowait, but we're not using full iops or bandwidth, I think, so not really sure. Maybe it's network?</p>



<a name="246668326"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20is%20timing%20out/near/246668326" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Joshua Nelson <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20is.20timing.20out.html#246668326">(Jul 21 2021 at 00:57)</a>:</h4>
<p>seems odd that it would be network if nginx is sending a response</p>



<a name="246668333"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20is%20timing%20out/near/246668333" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Joshua Nelson <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20is.20timing.20out.html#246668333">(Jul 21 2021 at 00:57)</a>:</h4>
<p>unless you mean that crater is fetching things from GCP to serve the request?</p>



<a name="246668353"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20is%20timing%20out/near/246668353" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20is.20timing.20out.html#246668353">(Jul 21 2021 at 00:57)</a>:</h4>
<p>no, e.g. log uploads to S3 could explain the i/o I <em>think</em></p>



<a name="246668393"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20is%20timing%20out/near/246668393" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20is.20timing.20out.html#246668393">(Jul 21 2021 at 00:58)</a>:</h4>
<p>not sure if iowait includes e.g. waiting on a tcp socket to flush or something like that</p>



<a name="246722140"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20is%20timing%20out/near/246722140" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Joshua Nelson <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20is.20timing.20out.html#246722140">(Jul 21 2021 at 13:51)</a>:</h4>
<p>Still down :(</p>



<a name="246729689"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20is%20timing%20out/near/246729689" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20is.20timing.20out.html#246729689">(Jul 21 2021 at 14:46)</a>:</h4>
<p>something is still pinning cpu in iowait, going to do a bit of investigation (or try)</p>



<a name="246731133"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20is%20timing%20out/near/246731133" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20is.20timing.20out.html#246731133">(Jul 21 2021 at 14:56)</a>:</h4>
<p><span class="user-mention" data-user-id="232545">@Joshua Nelson</span> ok, back up. I suspect something pins crater in reading <em>something</em> off network and disk at a sustained higher rate, but I don't really know what causes it</p>



<a name="246731161"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20is%20timing%20out/near/246731161" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20is.20timing.20out.html#246731161">(Jul 21 2021 at 14:56)</a>:</h4>
<p>restarting the server seems to fix it though.</p>



<a name="246732805"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20is%20timing%20out/near/246732805" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Joshua Nelson <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20is.20timing.20out.html#246732805">(Jul 21 2021 at 15:04)</a>:</h4>
<p>Thanks!</p>



<a name="246732864"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20is%20timing%20out/near/246732864" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Joshua Nelson <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20is.20timing.20out.html#246732864">(Jul 21 2021 at 15:05)</a>:</h4>
<p>Hmm, crater is showing the estimated time as a month? That seems wrong, is it just because it's at the start of the run and the estimates are inaccurate?</p>



<a name="246732880"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20is%20timing%20out/near/246732880" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Joshua Nelson <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20is.20timing.20out.html#246732880">(Jul 21 2021 at 15:05)</a>:</h4>
<p><a href="https://crater.rust-lang.org/ex/pr-87050">https://crater.rust-lang.org/ex/pr-87050</a></p>



<a name="246733143"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20is%20timing%20out/near/246733143" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20is.20timing.20out.html#246733143">(Jul 21 2021 at 15:07)</a>:</h4>
<p>estimates are always wrong</p>



<a name="246733158"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20is%20timing%20out/near/246733158" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20is.20timing.20out.html#246733158">(Jul 21 2021 at 15:07)</a>:</h4>
<p>should be no more than a couple days</p>



<hr><p>Last updated: Aug 07 2021 at 22:04 UTC</p>
</html>